3 Approximation Error vs . Estimation Error

A loss function is a mapping l : Y×Y "→ R (sometimes R×R "→ R). For example, in binary classification the 0/1 loss function l(y, p) = I(y ≠ p) is often used and in regression the squared error loss function l(y, p) = (y − p) is often used. Other loss functions include the following: absolute loss, Huber loss, εinsensitive loss, hinge loss, logistic loss, exponential loss, modified least squares loss, etc. They will be discussed later in more details.